STAT 


STAT 


Talorne! Hea 7 
Approved core llaunisieative, = “CHLRDPS6- 101 oot 200030003-8 


ODP-81-1154 
2 SEP 1981 


MEMORANDUM FOR: Director of Logistics 


FROM : Bruce T. Johnson 
Director of Data Processing 


SUBJECT : Utilities for ODP Computers 
REFERENCE : Memorandum for D/ODP from O/L, 


dtd 24 Aug 1981, Same Subject. 
(OL 13293a, ODP 81-1117) 


l. The referenced memorandum requests that we allow the 
int (OL, ODP, OC) committee currently addressing reliability of 
our utility system to complete its report prior to initiating any 
action to upgrade our utility performance. Of course, this is a 
reasonable request and we concur. We are looking forward to this 
committee promptly preparing its report to allow us to get on 
with constructing a solution to our utilities problems. On the 
subject of prime time repair of utilites, we recognize that there 
may be instances where this is inevitable. We ask, however, that 
every effort be made to schedule non-critical repairs to minimize 
the impact on Agency computer customers. We wish to work with 
your personnel on this matter. The ODP contact is[____] 
Chief, Configuration and Environmental Management 
Branch, Engineering Division. Steve may be reached on[L_ | 
(black). 


2. Finally, we appreciate your offer of a tour of ongoing 
utilities projects and would be very interested in 
participating. [  ———C.C | the ODP Executive Officer, 
would be the appropriate ODP individual to schedule the tour. 
Please have your staff contact George[——s—sédi ACO 
arrange a mutually agreeable date. 
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3. We in ODP appreciate your thoughtful response to our 
request for assistance. We have no doubt that the Office of 
Logistics is concerned about this problem and is working 
diligently on a solution. I am sure that continuing this 
dialogue will help us to develop the finest utilities support for 
Agency computer centers that technology and resources will 
allow. When the joint committee completes its study, I hope we 
can promptly arrange a meeting to discuss the implementation of 
the recommended solutions to our utilities problems. 


STAT 


Bruce . ohnson 
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UTILITIES RELIABILITY STUDY 


I. The Utilities Reliability Committee was established as an 


ad hoc group of representatives from the Office of Communi - 
cations (OC), Office of Data Processing (ODP), and the Office 
of Logistics(OL). The task assigned was to determine the 
utility reliability needs of the Agency's critical utilities 
consumers and to examine the present utilities systems ability 
to satisfy those needs. 

The committee consisted of: 


[sd Chief, Configuration and Environmental 


Management Branch (ODP/ED/CEMB) . 


[ chit, Engineering Staff, Technical 


Support Branch (OC/DND/TSB) 
[ Chief, Headquarters Engineering 
Branch (OL/RECD/HEB) 
fcueaseneiny each of the three offices. From time to time, 
others were involved in the committee processes as advisors 


and as participants in the absence of a designated member. 


II. The committee's activity was divided into three phases 
(Attachment A). Phase one consisted of identifying the custom- 
er's requirements and itemizing problem areas by systems. The 
requirements portion of this effort is shown in Attachment B. 


Accordingly, the following levels of utilities reliability were 
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considered acceptable: 
Office of Communications - four unscheduled outages 
| per year. 
Office of Data Processing - one unscheduled outage 


per year. 


A. <An outage for the purposes of this study is either a 
total outage or a rapid excursion beyond the equipment's 


utilities tolerances. 


B. The equipment used by the OC is adversely affected by 

a utility outage for the period of the outage plus an 
approximate one-half hour service restoration period. The 
OC suaanhent is apparently very hardy and, once restored to 


an operational mode, it operates relatively error free. 


C. The equipment used by ODP is adversely affected by 
a utility outage for the period of the outage plus a 
four- to six-hour service restoration period. The ODP 
equipment ie apparently less hardy than OC's and once 
restored to an operational mode, it is prone to further 


malfunctions for an additional two weeks. 


D. Both groups of equipment are also affected by slow 
excursions or by utilities services whose characteristics 
linger about the outer limits of the equipment specifi- 


cations. These conditions nsualty follow when eauinmest 


Pee , 
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changes are made and previously acceptable conditions are 
subsequently taxed just beyond reliable operational 


limits. 


Ili. The second phase consisted of examining the effectiveness 
of the present system of utilities support for this special 


equipment. 


A. The need to provide veguaant utilities services 
became a requirement beginning in the 1969-70 period. 
Utilities problems became severe and the Headquarters 
Engineering Branch was created to address the problems. 
The solutions began with the condition that critical 
utilities should be automatic, autonomus, and redundant. 
Because of the limited and centrally located equipment, 
the solutions to provide a motor-generator set to smooth 
transient electrical excursions, as well as provide 415 Hz 
power; to install groups of small chillers; ana to backup 
all of them with a dedicated emergency generator were 
effective. But the utilities requirement continued to 
grow. The motor-generator sets were either discarded or 
relegated to a backup position behind new Uninterruptible 
Power Systems (UPS). The number of these UPS's has grown 
to four with two more to be added by Project SAFE. The 
Single diesel generator of 2500 KW is now one of five 


similarly sized units. The redundant chillers are now 
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barely useful as requirements have increased. No backup 
chiller is provided for Project SAFE as space to install 
such a unit is basically unavailable. The recourse has 
been to provide both primary and redundant chilled water 
at the General Services Administration (GSA) Powerplant 
a quarter of a mile from the served equipment. The 
Situation has now gone full circle from initially 
isolating critical equipment from the routine levels 

of operation and maintenance and providing special 
redundancy to the present mode where the only economical 
and practical solution appears to be to rely on the GSA 


operated and maintained Powerplant. 


B. A second feature of this critical equipment expansion 
has been lack of accurate long-range planning. The speed 
of proliferation of electronic equipment was not foreseen. 
As a result, the utilities support was generally ac- 
complished on a short schedule to accommodate an urgent 
requirement. In such a: time dominated atmosphere, expedi- . 
ent design solves the immediate problem at the expense of 
long-range needs, including reliability. This comment is 
made in a constructive, informational way and is not in- 


tended to provoke a discussion on the virtues of hind- 


Sight. 


C. As the complexities of the utilities System increased, 


Lt Hieht be expected that the ana tess fic 
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system would expand also. After all, the basic reason 
for the first automatic autonomus and redundant systems 
was to provide an operation posture above that received 
from GSA equipment and manpower standards. Such was not 
the case as evidenced by programs to reduce personnel 
levels both by Agency managers as well as national adminis- 
trations. Doing more with less became the goal. Attempting 
to procure more service from GSA by offering reimbursement 
for operation and maintenance personnel for the emergency 
generators and special system preventive maintenance has 
been less than totally fruitful as GSA has gone through 
personnel reductions and is presently in a hiring freeze 


mode with a reduction in force rumored. 


D. In the private sector, the UPS's maintenance needs have 
been contracted solely to the respective UPS manufacturer 

to achieve maximum reliability. Here also, the level of 
effort has depended on the competence of the assigned 
technician except when the frequency of failures becomes 
sufficiently high to embarrass the company's management into 


sending a technically superior troubleshooter to solve the 


problem. 
E. The general evaluation of the present systems is detailed 


in Attachment C. The evaluation is naturally against the 


more strict one outage per year requirement of ODP. To 
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allow for a greater number of outages such as the four 


per year acceptable by OC could reasonably be expected 
Without significantly changing the present modes of utili- 


ties support. 


IV. The committee's recommendations are of necessity general. 


See Attachment D. 


A. Regardless of which courses of action are pursued, it 
was perceived that the future requirements for electronic 
equipment for the Agency are far reaching. Project SAFE 
will provide a new terminal system in the analyst's en- 
vironment. The backlog for present terminal installations 
requirements has multiplied by a factor of six, creating 
Project Speedwire with multiyear funding forecasts. The 
Information Handling Systems Architect/DDA has been created. 
As more daily work operations become computer dependent, the 
climate for reliable service will change from one of concern 
to one of severe or unacceptable impact. This need to provide 
extraordinary reliability will generate bieiecen coatiods 

on all phases of utilities support in concert with all 

other components in the end-to-end chain between the point 
of utility generation and the user's terminal. It seems, 
therefore, that more strict control in the form of modified 


configuration control is inevitable if the Agency continues 
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toward total reliance on automated data processing. 


B. To continue, as in the past, at funding consistent with 
current forecasts should allow utilities reliability to 
closely approach OC's GSiie-itas epee yeat minimum with the 
addition of a technical condition survey of the existing 
system. This survey is estimated to cost around $500,000 
and can be scheduled consistent with the Agency's desire 

to achieve increased reliability. To achieve ODP's one 
outage per year will require the initial survey followed 
by or incorporated in an annual maintenance and engineering 
effort of $2,000,000 to $8,000,000 to approach the conditions 
of a satellite ground station. Probably this effort would 
be a contractural arrangement with a company rather than 


with GSA. 


C. As a side issue related to overall reliability, but 
not necessarily dealing with utilities, a four to six hour 
period of computer down time historically occurs as the 
result of a short utilities malfunction. Should the com- 
puter industry develop more hardy hardware that is less 
susceptible to utilities anomalies, the utilities relia- 
bility requirements could paces less important. The 


present advances in hardware seem to be in reducing the 


7 7 “ 
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size and the electrical power consumption while increasing 
the computer power rather than in improving the hardiness 


or robustness against utilities anomilies. 


STAT 
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UTILITIES RELIABILITY AGENDA 


I. Identification of Problem Areas 
A. Itemize problem areas by utility system. 


B. Quantify "uptime" requirements in problem areas 
in terms of: 


1. percentage "uptime'' per year 
2. total outages per year 
3. excursions from load tolerance bandwidth 
4. scheduled outages versus unscheduled 
5. other appropriate parameters 

Il. Evaluation of present systems in terms of meeting 

' uptime requirements. 
A. Systems configuration. 


B. Systems hardware. 


III. Recommend changes. 
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REQUIREMENTS 


I. Utility Outages 
A. Scheduled - as necessary 
B. Unscheduled - not to exceed 1 hour in duration 
(1) OC - 4 per year per system 


(2) ODP - 1 per year per center (2 centers) 


II. Capacity Requirement - 1987 
A. ODP 


(1) 400 Hz: per attached schedule; all new 
computers at 40kVA 


(2) 60 Hz: 
a. GC03/1D16 complex 
1300 kVA including 2 frequency 


converters 


b. GC47 
500 KVA 


B. OC 
(1) 1100 kVA 60 Hz* 
€2) 50 kVA 400 Hz 
*A small portion may not impact A vault. 


III. Utility Tolerances - any excursion beyond these 
tolerances is considered an outage. 


A. Cooling 


(1) underfloor air 50 degrees to 65 degrees F 
50% to 70% R.H. 
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(2) 


(3) 


Power 


(1) 


(2) 


above floor air 70 degrees to 85 degrees F 


45% to 65% R.H. 


water cooling 45 degrees F to 60 degrees F 


28 to 38 GPM per processor 


no flow interruption of more than 3 minutes. 


60 Hz 
a. frequency plus minus 1/2 Hz 
b. steady state voltage 
208 volts + 6%, -8% 
Cc. transient voltage +15%3, -18% 
415 Hz 
a. frequency plus minus 20 Hz 
b. steady state voltage 
208 volts plus minus .2% 
c. transient voltage plus minus +8%, -15% 


half sec. 
-9 sec. recovery 
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Present System Evaluation 


To target for zero outages with the maximum tolerable 
limit being 1 outage per computer center, it will be 
necessary to configure and operate the utility system 
in a fashion similar to a satellite ground station. 
The present system fails to meet in following areas: 


A. 
B. 
C, 


Hardware maintenance and reliability 


Hardware system design — 


Operational control 


As a starting point the following general areas 
should be addressed: 


A. 


Hardware Maintenance and Reliability 


dy 
ie 


UPS are not performing to specification 


Level of utility maintenance is unacceptable 


Hardware System Design 


1. 
a 


400 Hz system design is inadequate 


VEPCO reliability varies, according to how 
and when VEPCO changes other system variables 


Raised floor clearances in GC0O3 are insufficient 
for the proper air flow 


Operational Control 


1 


GSA's performance in the area of operational 
control is non-existent. 


Changes in computer loads have not been 
coordinated closely enough. Ground stations 
generally employ some variation of configuration. 
control. At the present time, computer load 
projections have not allowed sufficient time 

for utility system changes (or conversely, 
utility systems have not changed fast enough 

to respond to new computer requirements). This 
results in time deadline dominated rather than 
reliability dominated utility systems designs. 
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ae. in addition, the respective roles of CEMB and 
HEB in utility system planning and epere Even 
need to be better defined. 
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Attachment 
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RECOMMENDATIONS 
Assumptions 
A. UPS's are necessary to meet tolerance requirements. 
B. There will be one 400 Hz and one 60 Hz UPS per ODP 
center fed from the same electrical source. 
C. The utility system expansion must start now to 
meet 1987 load projections. 
D. ODP and OC will remain on the south side of the 


building as load/space requirements increase. 


Possible Hardware Changes 


A.” 


B. 


C. 


Bring UPS up to specification. 
Redesign 400 Hz system. 


Investigate reconnection to VEPCO or otherwise 
increase, VEPCO reliability. 


Provide new southside power vault and UPS. 


Provide additional northside electrical feeder. 


‘Provide HVAC enhancement in GC03 through: 


1. removing old cable, controlling installation 
of’ new cable. 


2. providing more air handlers. 
3. raising floor. 


4. opting for computer communications that require 
less cable. 


5. all of above. 
Replace 3 chillers at Powerplant. 
Add air handlers and increase C.W. piping sizes to 


both ODP and OC for projected load increase and 
redundancy. 
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I. Add more back-up generating capacity. 
J. Replace C.W. lines from Powerplant. 
III. Possible Maintenance Changes | 


Eliminate or upgrade GSA effort. Catch up on extensive 
backlog of "one outage" maintenance items. 


IV. Possible Operational Changes 


A. Provide better and more regular computer load 
projections. 


B. Consider "overexpanding" the utility system. This 
would minimize facilities considerations in 
computer system planning. 


C. Provide some type of configuration control, though 
not as formal as used at some ground stations, as 
an alternative to IV.B. above. 


V. Other Considerations 


A. Investigate the relaxation of outage limitation. 
(Perhaps 4 per year?) 


B. Removing the computers to another building which 


would be designed and operated in the 
high reliability mode of a ground station. 
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